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SECTION  I 


INTRODUCTION 


PURPOSE 


This  report  discusses  a variety  of  questions  concerning  the  appl irabil ity, 
methods , and  philosophy  of  the  NAVAIR/NAVTRAEQU IPCEN  model  for  large-si.  e In- 
structional Systems  Development  (ISO).  Although  many  of  the  questions  raised 
in  this  report  apply  generally  to  other  training  system  models  currently  in  use 
throughout  the  armed  services  and  industry,  the  Intent  here  has  been  to  address 
only  those  matters  of  concern  that  derive  directly  from  the  NAVAIR/NAVTRAEQU IP- 
CEN model  as  It  is  presently  formulated  in  the  specification  (MIl-T-29053) 
entitled  "Training  Requirements  for  Aviation  Weapon  Systems"  (October  1977), 
and  in  the  associated  Data  Item  Descriptions  (DIDs).  Even  though  the  scope  of 
this  report  is  limited  to  the  NAVAIR/NAVTRAEQU IPCEN  model,  it  should  not  be 
assumed  that  all  areas  open  to  valid  and  important  inquiry  have  been  exhausted; 
neither  should  the  questions  raised  here  be  regarded  as  a definitive  evaluation 
of  the  current  NAVAIR/NAVTRAEQU IPCEN  model. 

The  model  has  already  undergone  several  revisions  subject  to  knowledge 
gained  from  various  applications  of  it  to  both  existing  and  newly  emerging 
weapon  systems  training  programs.  By  continuing  this  evaluation  process  the 
applicability  of  the  model  can  be  refined  and  maintained  in  a dynamic  state, 
and  its  responsiveness  to  the  training  needs  in  the  Naval  aviation  community 
can  be  augmented. 

The  purpose  of  the  present  report  is  to  extend  the  on-going  evaluation  of 
the  NAVAIR/NAVTRAEQU IPCEN  model  beyond  the  level  of  specific  application- 
oriented  problems  to  a more  theoretical  level  of  fundamental  questions.  The 
questions  discussed  in  this  report  were  viewed  as  items  offering  major  poten- 
tial for  fruitful  consideration  in  the  course  of  future  development  and  appli- 
cation of  the  model. 

HISTORY 

Growth  in  the  Inherent  complexity  of  airborne  weapon  systems  over  the  last 
thirty-odd  years  has  been  paralleled  by  the  emergence  of  an  equally  complex 
problem  - the  design  of  large-scale  Instructional  systems.  Until  recently  the 
Navy’s  response  to  this  problem  has  been  much  the  same  as  the  other  armed 
services  and  industry.  The  concept  of  systems  analysis  was  embraced  as  an 
operating  strategy  and  the  principles  of  applied  psychology  and  educational 
technology  were  transformed  Into  procedural ized  methodologies  for  training 
program  development.  The  result  was  called  the  "Systems  Approach  to  Training" 
(SAT). 

The  principles  underlying  SAT  were  sound,  but  the  approach  suffered  a 
fundamental  weakness.  It  lacked  a management  system  capable  of  directing 
multiple  applications  of  SAT,  i.e.,  a general  control  process  that  would  effect 
a uniform  application  of  SAT  across  training  programs  developed  for  different 
weapon  systems. 
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By  the  mid-1970s  the  Naval  Training  Equipment  Center  ( NAVTRAEQU I PC EN ) , 
acting  under  the  auspices  of  the  Naval  Air  Systems  Conmand  (NAVAIRSYSCOM) , had 
undertaken  the  task  of  constructing  a generally  applicable  model  for  ISO.  The 
ISO  process  was  divided  Into  five  major  phases  of  activity  - analysis,  design, 
development,  Implementation,  and  quality  control.  The  activities  to  be  carried 
out  within  each  major  phase  were  Identified  and  further  developed  Into  logi- 
cally coherent  procedures  that  made  explicit  the  key  decision  points,  specific 
objectives,  and  end-products  required  at  each  stage  In  the  process.  The  re- 
sulting standardization  of  the  ISO  process  made  possible  effective  management 
control  throughout  the  successive  stages  of  each  developmental  project,  and  It 
permitted  cross-project  comparisons  essential  to  further  refinement  and  gener- 
alization of  the  process. 

The  NAVAIR/NAVTRAEQUIPCEN  model  of  the  ISO  process  has  been  widely 
applied  In  the  development  of  a variety  of  training  programs  throughout  Naval 
aviation,  and  the  model  has  undergone  several  revisions  In  response  to  feed- 
back from  these  applications.  As  a management  tool,  the  model  has  proved  Its 
value,  but  It  continues  to  be  evaluated  as  experience  with  It  accumulates. 

OVERVIEW 

Seven  major  areas  of  ISO  were  considered  to  be  of  sufficient  practical 
and  theoretical  Interest  to  merit  Inquiry  In  terms  of  the  NAVAIR/NAVTRAEQUIPCEN 
model.  These  areas  are:  (1)  subject  matter  expert  (SME)  training  and  utili- 
zation; (2)  determination  of  time-phased  segmentation  of  the  ISD  process; 

(3)  Instructional  media;  (4)  tralnlnq  device  design  and  utilization;  (5)  In- 
structor training  and  continuity;  (6)  task  analysis,  listings,  validation,  and 
selection;  (7)  behavioral  objectives  and  hierarchies.  Within  each  of  these 
areas  a number  of  subtopics  were  selected  for  discussion.  The  problematic 
nature  of  each  subtopic  Is  Identified  within  the  context  of  the  NAVAIR/NAVTRA- 
EQUIPCEN model,  focused  Into  the  format  of  a question,  and  suggested  approach- 
es are  offered  that  may  be  helpful  In  obtaining  the  desired  answer  to  each 
question.  Subtopics  that  relate  to  others  in  the  text  are  appropriately 
cross-referenced.  Each  major  area  In  this  report  Is  referenced  to  the  appli- 
cable sectlon(s)  of  the  model  (MIL-T-29053)  as  well  as  to  the  pertinent  DID ( s) . 


4 


4 


l 

\ 


NAVTRAEQU IPCEN  IH-309 
SECTION  II 

SUBJECT  MAHER  EXPERT  TRAINING  AND  UTILIZATION 

The  questions  raised  here  are  referenced  primarily  to  Data  Item  Descrip- 
tion number  UDI -H-257 1 2 (Subject  Matter  Expert  Training  Materials),  and  to 
military  specification  number  MIL-T-29053,  Training  Requirements  for  Aviation 
Weapon  Systems,  sections  3.1,  3.3,  3.7,  3.9,  3.10,  3.14,  3.17,  3.18,  3.20,  and 
3.21. 

SURVEY  OF  SME  PROBLEMS 

Since  the  success  of  an  ISD  project  depends  so  critically  upon  the  proper 
functioning  of  SMEs,  especially  during  the  early  stages,  it  would  seem  worth- 
while to  undertake  a program  aimed  at  determining  the  kinds  of  problems  pre- 
viously encountered  in  the  use  of  SMEs.  What  have  these  problems  been  from 
the  point  of  view  of  the  contractor,  on  the  one  hand,  and  what  have  they  been 
from  the  point  of  view  of  the  SMEs?  The  data  necessary  to  answer  these 
questions  could  be  obtained  by  means  of  questionnaires  that  would  be  sent  to 
the  contractors  who  have  in  the  past  worked  with  SMEs,  as  well  as  to  the  SMEs 
who  have  participated  in  ISD  projects.  Different  forms  of  the  questionnaire 
would  need  to  be  prepared  for  the  SMEs  and  for  the  contractors.  Some  of  the 
problems  that  could  be  Identified  by  this  means  would  be  peculiar  to  the 
particular  ISD  program,  but  the  thrust  of  this  effort  would  be  to  abstract 
from  the  questionnaires  from  all  programs  any  problems  that  they  all  had  in 
common.  For  example,  SME  motivation,  conflicts  in  personalities,  communica- 
tion, difficulties  with  certain  areas  or  requirements  in  the  specs,  etc. 

SME  EXPERTISE  IN  ISD 

The  DID  dealing  with  Subject  Matter  Expert  Training  Materials,  ninber 
UDI-H-25712,  stipulates  that  the  contractor  will  prepare  a systematically 
developed  package  for  SME  training.  The  stipulations  laid  out  in  the  DID 
appear  complete  in  that  they  cover  all  areas  of  the  ISD  process  in  which  SMEs 
are  involved.  Essentially,  the  DID  stipulates  to  the  contractor  what  areas 
the  contractor's  training  package  for  SMEs  should  include.  However,  the  ex- 
tensiveness of  the  SME  training  program  must  depend  to  some  extent  on  the  pre- 
vious experience  of  SMEs  with  training  programs  in  general,  and  ISD  analyses 
in  particular.  It  Is  probably  safe  to  assume  that  entry  level  of  SMEs  with 
regard  to  ISD  analyses  are  zero.  Thus,  the  extensiveness  of  the  contractor's 
SME  training  package  will  be  governed  by  two  things,  the  time  alloted  for  the 
SME  training,  and  the  degree  of  ISD  expertise  that  SMEs  must  achieve.  It 
would  seem  reasonable  to  assume  that,  in  any  case,  SME  training  would  have  to 
be  brief.  This  would  mean  not  only  that  SME  training  would  have  to  be  highly 
efficient,  but  also  that  the  level  of  ISD  expertise  would  have  to  be  somevrtiat 
attenuated.  Thus,  it  would  be  important  that  the  contractor  know  the  answer 
to  the  following  question;  what  level  of  expertise  must  SMEs  achieve  in  each 
of  the  areas  of  activity  stipulated  in  the  DID  dealing  with  subject  matter  ex- 
pert training  materials?  SME  expertise  level  could  be  specified  in  terms  of 
the  following:  (1)  a task  analysis  and  listing  for  SME  activities  in  the  ISD 
process;  (2)  an  analysis  and  statement  of  the  degree  of  conceptual  compre- 
hension of  abstract  ISD  principles  needed  by  SMEs;  (3)  an  identification  of 
areas  of  ISD  activity  which  have  proved  in  the  past  to  be  especially  problemat- 
ical for  SMEs  to  learn. 
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TIME  FLOW  OF  SME  TRAINING  AND  UTIL  1 TAT  ION 

While  the  DIO  concerning  Subject  Matter  Expert  Training  Materials  is 
quite  complete  regarding  areas  in  which  SMEs  anp  to  be  trained,  it  sa.vs 
nothing  regarding  such  matters  as  optimum  procedures  for  obtaining  the  re- 
quired SME  performance  in  those  areas,  nor  does  it  state  the  optimum  number  of 
SMEs  required  to  achieve  the  objective  laid  out.  For  example,  what  is  the 
-most  effective  way  to  achieve  a task  listing?  How  many  SMEs  does  a contractor 
need  in  order  to  carry  out  a task  listing?  Is  there  a critical  number  of  SMEs 
which  will  be  optimally  effective  for  achieving  each  objective,  a number  above 
which  communication  and  effectiveness  will  break  down,  and  below  which  objec- 
tive realization  becomes  impeded?  Is  it  necessary  that  all  SMEs  be  trained 
in,  and/or  be  utilized  in  all  phases  of  the  ISD  process  calling  for  SMEs? 
F*erhaps  a division  of  SME  responslbil  ities  would  be  more  efficient  since  this 
would  not  retire  that  all  SMEs  be  trained  in  all  areas,  and  this  might  allow 
differently  trained  SMEs  to  work  on  different  assignments  simultaneously.  Or, 
from  a slightly  different  perspective,  a small  group  of  SMEs  might  be  trained 
to  carry  out  first -stage  tasks,  and  then,  at  the  point  in  time  when  this  first 
group  began  work,  training  of  the  second  group  of  SMEs  on  the  part  of  the 
project  for  which  they  would  be  responsible  could  begin,  and  so  on.  This  type 
of  time-flow  in  SME  training  and  utilization  might  not  only  be  more  efficient, 
but  also  it  might  avoid  problems  of  SME  turnover,  etc.  Of  course,  this  raises 
another  question,  to  what  extent  may  each  SME  activity  be  trained  independent- 
ly? For  example,  must  SMEs  working  on  objective  hierarchies  also  be  trained 
in  task  listing?  Division  among  certain  areas  of  SME  activity  probably  will 
not  be  possible.  However,  because  of  the  potential  gain  by  taking  this  ap- 
proach, it  would  be  highly  desirable  to  determine  what  areas  of  SME  activity 
may  be  trained  independently. 

QUALIFICATIONS  AND  PERSONALITY  OF  THE  ISD  EXPERT 

Since  the  contractor's  representative  responsible  for  training  SMEs  and 
guiding  them  through  the  successive  stages  of  the  ISD  process  will  probably  be 
a psychologist  with  limited  military  experience,  if  not  limited  ISD  experience, 
ft  would  seem  worthwhile  to  inquire  into  the  matter  of  qualifications  of  the 
presumed  ISD  expert.  Assuming  that  the  contractor  can  be  called  upon  to  pro- 
vide psychologists  with  sufficient  ISD  experience,  there  still  remains  the 
problem  of  the  kind  of  relationship  that  this  individual  will  be  able  to  estab- 
lish and  maintain  with  the  SMEs.  It  is  probably  not  an  overstatement  to  say 
that,  to  a large  degree,  success  of  the  ISD  project  depends  upon  the  kind  of 
relationship  that  will  exist  between  the  psychologist  and  the  SMEs.  Thus,  it 
would  seem  important  to  know  what  kind  of  relationship  will  be  most  effective 
and  productive,  and  what  personality  characteristics  the  psychologist  needs  in 
order  to  maximize  the  probability  of  achieving  this  relationship?  Essentially, 
the  question  is,  what  array  of  social  behaviors  will  be  optimal  in  establishing 
and  maintaining  an  effective  and  productive  relationship  with  military-trained 
subject  matter  experts?  This  question  could  probably  be  answered  sufficiently 
well  to  enable  a general  specification  to  contractors  which  would  be  helpful  to 
them  as  an  aid  in  selecting  their  representative. 


6 


r 


IH 


A 


1 


[ < 


t 


NAVTRAEQUIPCEN  IH-309 
SECTION  III 

DETERMINATION  OF  TIME-PHASED  SEGMENTATION  OF  THE  ISD  PROCESS 

The  questions  raised  in  this  section  are  referenced  primarily  to  Data 
Item  Description  number  UDI-H-25710  (Work  Plan),  and  Data  Item  Description 
number  UDI-H-25711  (Progress  Report),  and  to  the  document  entitled  Training 
Requirements  for  Aviation  Weapon  Systems  (MIL-T-29053). 

ISD  PROCESS  SEGMENTATION 

Time-phased  segmentation  of  the  ISD  process  offers  many  potential  advan- 
tages to  both  the  Navy  and  the  contractor.  Two  of  the  more  prominent  advan- 
tages of  time-phased  segmentation  are;  (1)  contracts  could  be  awarded  for  seg- 
ments of  the  ISD  process  rather  than  the  whole  thing,  and  (2)  the  proposed 
work  plans  and  budgets  of  potential  contractors  could  be  prepared  and  evalu- 
ated more  accurately  with  respect  to  projected  time,  manpower,  cost,  re- 
sources, etc.  For  example,  it  might  be  advantageous  for  the  Navy  to  use 
different  contractors  on  different  segments  of  the  ISD  process.  Due  to 
differences  in  expertise,  resources,  etc.,  between  contractors,  one  contractor 
might  be  better  prepared  to  carry  out  a particular  segment  than  another  con- 
tractor. Also,  for  a variety  of  reasons,  it  might  be  necessary  for  the  Navy 
to  terminate  a contract  at  the  end  of  a particular  segment  and  to  have  a 
different  contractor  continue  the  ISD  project  from  that  point  forward.  On  the 
other  hand,  small  contractors  with  considerable  expertise  in  particular  areas 
might  be  inclined  to  bid  for  contracts  if  their  responsibil  ity  were  limited  to 
a particular  and  highly  defined  segment  of  activity.  In  order  for  any  of  this 
to  be  possible,  the  ISD  process  must  be  broken  down  into  mutually  exclusive, 
non-over-1  apping  segments  that  are  time-phased.  Thus,  the  question  is:  What 
portions  of  the  ISD  process  can  be  broken  down  into  self-contained  segments 
of  activity  which  have  a defined  starting  point,  specified  objectives, 
products,  manpower  and  resources  requirements,  and  time  rate  of  progress  of 
completed  ISD  projects  might  provide  the  answer  to  this  question. 

SIMULTANEOUS  COMPLETION  OF  DIFFERENT  SEGMENTS  OF  THE  ISD  PROCESS 

This  question  is  really  a subtopic  of  the  foregoing  question,  but  its 
potential  importance  would  seem  to  warrant  treating  it  separately.  Should  it 
be  possible  for  the  ISD  process  to  be  broken  down  into  self-contained,  inde- 
pendent segments  that  were  properly  time-phased  with  respect  to  one  another, 
it  might  be  possible  to  shorten  the  time  to  completion  of  the  entire  ISD 
project  by  having  different  segments  completed  simultaneously  by  different 
contractors.  The  question  is:  Which  segments  could  be  carried  out  simulta- 
neously? Obviously,  in  order  that  any  two  segments  be  carried  out  simulta- 
neously, the  products  of  neither  one  could  serve  as  prerequisites  of  the  other. 
That  is,  there  could  be  no  substantial  sequential  dependencies  between  any  two 
segments  if  they  were  to  be  carried  out  simultaneously.  As  a part  of  the  study 
suggested  in  the  above  question,  sequential  dependencies  between  separate  seg- 
ments could  be  identified.  If  they  should  prove  to  be  minor,  a strategy  prob- 
ably could  be  developed  to  eliminate  them.  The  objective  would  be  to  reduce 
to  a minimum  the  number  of  sequentially  dependent  segments,  thereby  increasing 
the  number  of  segments  that  could  be  completed  simultaneously. 
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SECTION  IV 
INSTRUCTIONAL  MEDIA 

The  questions  raised  in  this  section  are  referenced  to  Data  Item  Descrip- 
tion number  UDI-H-25718  (Media  Selection  Model),  Data  Item  Description  number 
UDI-H-25722  (Training  Support  Requirements  Analysis)  and  Data  Item  Description 
number  U DI-H-257 1 9 (Media  Selection  Report),  and  to  sections  3.9,  3.13,  and 
6.4  of  Training  Requirements  for  Aviation  Weapon  Systems  (M IL-T-29053 ) . 

THE  MEDIA  SELECTION  MODEL 

The  process  of  selecting  instructional  media  that  are  optimally  matched 
to  behavioral  objectives  is  as  complex  as  it  is  critical  to  the  success  of  an 
ISD  project.  The  importance  of  media  selection  is  reflected  in  both  the  DIDs 
and  specifications  where  it  is  required  of  the  contractor  that  an  explicit 
media  selection  model  be  provided.  An  example  of  such  a model  and  its  use  is 
provided  in  Section  6.4  of  the  specifications.  The  model  serves  as  a mechanism 
for  determining  first,  second,  third,  and  fourth  choices  of  media  for  each 
behavioral  objective.  The  user  begins  by  answering  each  of  five  questions 
about  a specific  behavioral  objective.  For  each  question  the  user  must  decide 
which  of  several  coded  answers  is  most  applicable  to  the  particular  behavioral 
objective  in  question.  The  coded  answers  to  these  five  questions  can  then  be 
used  to  step  through  the  decision  matrix  stipulated  in  the  model.  There  are 
44  terminal  points  in  this  decision  matrix  and  each  possible  pathway  through 
the  matrix  leads  the  user  to  one  of  these  terminal  points.  Upon  arriving  at 
a terminal  point,  the  user  then  turns  to  a table  which  enumerates  the  rank- 
ordered  choices  of  media  for  each  terminal  point.  Thus,  the  model  stipulates 
a rational  decision  process  for  determining  optimal  media  selection.  However, 
the  success  of  the  model  depends  almost  entirely  upon  the  nature  of  the  initial 
questions  one  must  ask  about  each  behavioral  objective,  and  the  theoretical 
significance  of  the  kinds  of  distinctions  made  in  the  coded  answers  to  each  of 
these  questions.  The  magnitude  of  importance  of  the  outcome  from  this  model 
is  entirely  too  great  to  merely  assume  that  the  initial  questions  and  distinc- 
tions characterizing  the  answers  to  these  questions  are  in  fact  the  best 
possible  set  of  questions  and  distinctions.  The  basic  question  being  raised 
here  is  both  empirical  and  theoretical.  With  what  degree  of  confidence  can  we 
assume  that  these  five  questions  and  the  distinctions  they  offer  as  answers  are 
either  appropriate  or  sufficient?  Even  though  the  specifications  and  DIDs  call 
upon  the  contractor  to  provide  their  own  decision  models  for  media  selection, 
the  importance  of  this  subject  would  seem  to  warrant  an  independent  study. 

Such  a study  should  be  carried  out  by  someone  expert  in  both  learning  theory 
and  instructional  media.  Part  of  the  study  would  be  aimed  at  establishing  a 
kind  of  catalog  of  currently  available  instructional  media,  the  kinds  of  in- 
formation display  and  presentation  utilized  by  each  medium,  and  the  kinds  of 
behavior  such  medium  is  most  capable  of  training.  Once  the  catalog  was  estab- 
lished, an  experimental  program  could  be  undertaken  with  those  media  which  were 
judged  to  be  most  appropriate  within  the  military  framework.  The  purpose  of 
this  experimental  study  would  be  to  determine  not  only  the  limits  of  each 
medium,  but  the  relevant  behavioral  dimensions  most  appropriate  for  use  in  a 
media  selection  model.  A further  outcome  of  this  kind  of  inquiry  might  be  a 
kind  of  matrix  in  which  behavioral  dimensions  on  the  one  hand  would  be  related 
to  media  types  on  the  other  hand.  At  the  conjunctions  of  behavioral  dimensions 
and  each  media  type  numerical  indexes  representing  relative  cost-effectiveness 
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question  that  an  attempt  would  be  made  to  establish  the  validity  of  media  types 
with  respect  to  such  performance  variables  as  rate  of  acquisition,  recall. 


discrimination,  appropriateness  of  rule  use,  etc.,  and  the  long  term  reliabil- 
ity of  criterion  level  performance.  As  part  of  this  last  objective,  the 


researcher  would  attempt  to  identify  the  sources  and  kinds  of  incentives  that 
could,  or  would,  be  associated  with  each  media  type. 


NEW  KINDS  OF  INSTRUCTIONAL  MEDIA 

Although  great  strides  have  been  made  in  all  aspects  of  instructional 
media  over  the  past  25  years,  we  are  still  faced  with  the  fact  than  an  expertly 
prepared  workbook  will,  for  many  kinds  of  instructional  objectives,  be  nearly 
as  effective  as  any  other  more  costly  instructional  medium.  With  the  exception 
of  training  in  perceptual -mo tor  skills,  which  may  require  sophisticated  weapon 
systems  trainers,  it  appears  that  the  old-time  programmed  workbook  falls  short 
of  its  modern  electronic  equivalents  only  in  the  area  of  motivational  incen- 
tives. And  whil e motivational  incentives  certainly  provide  the  juice  that 
keeps  the  cognitive  wheels  oiled,  as  it  were,  one  is  nevertheless  left  with 
the  impression  after  viewing  a professionally  prepared  videotape  presentation 
that  the  instructional  value  of  the  show  would  be  considerably  diminished  had 
its  viewer  not  been  provided  with  a well-prepared  workbook  as  a backup.  Thus, 
it  appears  that  we  are  still  in  the  horse-and-buggy  stage  of  instructional 
media.  In  the  opinion  of  this  writer,  the  fundamental  question  is:  What  is 
the  most  effective  way  to  present  information  and  require  a response  to  it  such 
that  the  result  will  be  competent,  enduring,  and  reliable  utilization  of  that 
information?  Optimally  we  would  like  to  achieve  training  methods  in  which 
rapidity  of  acquisition  would  not  trade  against  endurance  of  retention,  regard- 
less of  the  difficulty  of  the  subject  matter.  The  enormous  amount  of  time  re- 
quired by  humans  to  learn  complex  response  sequences  with  heavy  memory  loading 
is  an  example  of  a primitive  training  methodology  or  a wrong  approach  to 
training.  In  this  writer's  opinion,  it  is  the  latter.  In  spite  of  all  of  the 
research  which  has  been  conducted  on  learning  and  training  methodologies  over 
the  past  25  years,  relatively  little  attention  has  been  given  to  the  question: 
What  kind  of  functional  unit  does  the  human  represent  when  he  must  operate 
within  the  context  of  a specified  information-processing  and  action  system? 
Without  trying  to  answer  this  question  here,  it  would  seem  evident  that  the 
human  might  be  regarded  as  a decision-making  and  actuating  device  with  excep- 
tional perceptual  capabilities,  but  with  remarkably  poor  memory  capacity.  Such 
a concept  may  or  may  not  be  accurate.  The  point  is,  if  we  are  to  effectively 
train  men  to  function  as  units  in  sophisticated  information  processing  and 
action  systens,  then  our  concept  of  the  role  of  humans  operating  in  such  systems 
should  be  an  accurate  one.  Such  a concept  would  guide  both  the  design  of  the 
systems  men  must  operate  and  the  instructional  methodology  by  means  of  which  we 
train  them.  Ironically,  probably  through  trial -and-error  and  the  availability 
of  micro-electronics,  the  more  complex  systems  requiring  human  operators  seem 
to  have  been  designed  with  some  such  concept  in  mind,  while  the  media  selected 
for  training  men  to  operate  such  systems  seem  not  to  have  been  driven  by  any 
unified  concept  of  man  as  an  operational  unit.  The  relatively  greater  effec- 
tiveness of  weapon  systems  trainers  probably  stems  from  the  fact  that,  unlike 
videotapes  and  workbooks,  they  require  men  to  take  action  based  on  information 
stored  elsewhere  and  presented  in  the  proper  sequence  to  them.  And  so  we  find 
that  the  success  of  the  weapon  systems  trainer  is  not  so  much  in  its  fidelity 
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of  simulation,  but  rather  It  derives  from  the  manner  In  which  it  requires 
humans  to  utilize  Information  and  take  appropriate  action.  Extending  this  con- 

npt  to  the  usual  squadron  learning  center  with  carousels  and  associated  media 
brary,  one  can  readily  conceive  of  a centralized  system  driven  by  a computer 
which  would  (1)  provide  visual  and  auditory  displays  to  students  In  carousels, 
process  their  responses  to  Inputs  and  provide  feedback  Information,  (2)  provide 
random  access  to  any  segment  of  the  program  and  highly  individualized  Instruc- 
tion, and  (3)  simulate  In  a very  limited  way  the  actual  operational  situation. 
In  other  words,  a set  of  carousels  in  a learning  center  each  of  which  would  be 
a kind  of  mini-trainer  with  response  consoles  and  visual  and  auditory  displays 
driven  Independently  by  a centralized  computer  system.  This  Is  just  one  sort 
of  possibility  that  the  concept  discussed  above  might  lead  to.  The  point  of 
this  question  Is  that  research  needs  to  be  done  on  the  emerging  functional  role 
of  man  In  complex  operational  systems  as  a means  of  Identifying  the  behavioral 
dimensions  necessary  for  the  development  of  new  kinds  of  instruction  media. 

(See  Section  V (A  New  Training  Device  Concept).) 
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SECTION  V 

TRAINING  DEVICE  DESIGN  AND  UTILIZATION 

The  questions  raised  here  relate  to  Data  Item  Description  number  UDI-H- 
25716,  Data  Item  Description  number  UDI-H-25720,  and  to  sections  3.5,  3.6,  and 
3.10.1.2  of  the  Document  Training  Requirements  for  Aviation  Weapon  Systems 
(MIL-T-29053). 

TRAINER  MODIFICATION 

In  the  case  of  ISO  for  existing  weapon  systems,  the  specifications  recom- 
mend an  inventory  of  potential  training  resources  including  training  devices 
which  might  be  modifiable.  The  rationale  for  modifying  an  existing  training 
device  for  use  in  a new  training  program  is  that  the  cost  of  such  a modifica- 
tion will  be  less  than  the  cost  of  a new  training  device.  The  modifications 
made  are  those  required  by  the  training  objectives  of  the  new  ISD  program.  The 
operating  principle  that  comes  across  in  the  specs  might  be  stated  as  follows: 
Use  training  devices  that  are  available  if  they  can  be  modified  to  meet  the  new 
training  objectives.  Again,  the  driving  force  behind  this  emphasis  appears  to 
be  cost  savings.  While  cost  savings  will  always  appear  to  be  a highly  desir- 
able goal,  the  emphasis  in  the  specs  on  use  of  modifiable  training  devices 
appears  to  be  too  strong.  The  question  is  what  sort  of  training  device  would 
be  optimally  designed  if  only  the  behavioral  objectives  to  be  trained  were  con- 
sidered in  the  absence  of  the  constraint  imposed  by  the  necessity  to  utilize, 
if  possible,  an  already  existing  training  device?  More  directly,  how  would  an 
ideally  designed  training  device  compare  with  a modified  training  device? 
Presumably,  the  optimally  designed  device  would  result  in  a superior  realiza- 
tion of  the  training  objectives  than  the  modified  device.  In  fact,  the  best 
way  to  evaluate  the  modified  training  device  would  probably  be  to  compare  it 
with  the  optimally-designed  device.  This  does  not  mean  that  the  optimally- 
designed  training  device  would  actually  have  to  be  either  built  or  designed, 
only  that  its  characteristics  be  specified.  The  question,  then,  that  is  being 
raised  here  is:  How  does  one  (1)  determine  whether  a training  device  can  be 
modified  in  such  a way  as  to  realize  the  training  objectives,  and  (2)  what  is 
the  best  way  to  determine  the  needed  modification?  It  seems  to  be  implicit  in 
the  specs  that  all  one  needs  in  order  to  answer  these  two  questions  is  the  be- 
havioral objectives.  While  this  is  questionable,  the  specs  do  not  recommend  a 
procedure  for  answering  these  two  questions.  As  a matter  for  further  investi- 
gation, it  is  suggested  here  that  a good  procedure  would  be  to  deduce  from 
those  behavioral  objectives  that  require  training  device  utilization  the  ideal 
characteristics  of  the  optimal  training  device.  These  optimal  characteristics 
could  then  serve  as  a model  by  means  of  which  existign  training  devices  could 
be  assessed  for  possible  modification,  and,  if  modification  is  judged  to  be 
feasible,  the  extent  and  direction  of  modification  could  be  guided.  It  would 
thus  seem  to  be  worthwhile  to  develop  procedures  by  means  of  which  optimal 
characteristics  of  training  devices,  given  certain  behavioral  objectives,  could 
be  generated.  Research  in  this  area  could  lead  to  the  development  of  a set  of 
procedures  which  could  then  be  incorporated  into  the  ISD  specifications. 
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A NEW  TRAINING  DEVICE  CONCEPT 

It  would  appear  that  the  chief  restriction  on  utilization  of  training 
devices  is  the  cost.  Assuming  that  there  Is  a nearly  direct  relationship  be- 
tween training  device  cost  and  the  fidelity  of  simulation  built  into  the  de- 
vice, It  should  be  possible  to  reduce  costs  of  training  devices  by  reducing 
the  fidelity  of  their  simulation.  This  immediately  raises  the  question:  What 
degree  of  fidelity  of  simulation  ought  to  be  incorporated  into  training  de- 
vices? The  answer  to  this  question  would  seem  to  depend  almost  totally  on  the 
training  objectives  for  which  the  device  is  built.  In  the  ISO  specifications, 
training  devices  are  used  for  "hands-on"  objectives,  and  other  media  are  speci- 
fied for  the  remaining  objectives.  This  restriction  on  the  use  of  training 
devices  appears  to  be  the  result  of  their  very  limited  availability  vrfiich,  in 
turn,  reflects  the  enormous  cost  of  developing,  producing,  and  operating  a 
training  device.  Thus,  we  have  a situation  in  which  the  utilization  of 
training  devices  in  training  programs  is  controlled  by  their  cost-availabi 1 ity 
rather  than  their  potential  for  training.  Thus,  cost  of  training  devices  Is  a 
function  of  their  fidelity  of  simulation,  and  the  emphasis  on  simulation 
appears  to  be  centered  in  the  age-old  assumption  that  training  device  effec- 
tiveness must  be  measured  in  terms  of  transfer  of  training.  To  be  sure,  if  the 
fidelity  of  simulation  in  a trainer  nearly  matches  that  of  the  operational  en- 
vironment, then  transfer  of  training  is  a meaningful  criterion  of  training  ef- 
fectiveness. On  the  other  hand,  no  one  would  argue  that  transfer  of  training 
was  a meaningful  criterion  for  evaluating  the  effectiveness  of  training  carried 
out  with  workbooks  or  videotape  displays.  Effectiveness  of  training  produced 
by  the  latter  is  measured  In  terms  of  performance  criteria  derived  directly 
from  the  behavioral  objectives  being  trained.  In  general,  it  may  be  said  that 
it  is  the  behavior  that  is  trained  that  is  tested,  and  it  Is  the  test  perform- 
ance which  provides  the  basis  for  evaluation  of  the  effectiveness  of  training. 
If  fidelity  of  simulation  in  the  training  situation  is  high,  then  the  behavior 
to  be  tested  approximatees  that  required  In  the  operational  situation  and  the 
test  performance  may  be  taken  as  an  indication  of  transfer  of  training.  So, 
the  real  question  Is  not  whether  transfer  of  training  is  high,  but  whether, 
after  training  of  numerous  individual  behaviors,  performance  in  the  operational 
situation  will  be  at  criterion  levels.  This  is  not  to  say  that  high  fidelity 
trainers  are  not  Important.  Such  devices  probably  are  the  best  sort  of  Inter- 
face between  school -type  training  and  the  operational  environment.  The  point 
here  is  simply  that  most  training  is  not  carried  out  under  high  fidelity  con- 
ditions and,  therefore,  it  should  be  possible  to  design  and  utilize  low- 
fidelity,  low-cost  training  devices  which  would  have  the  potential  of  proving 
far  superior  to  the  traditional  workbooks,  videotapes,  etc.  Such  devices 
might,  for  example,  take  the  form  of  Individualized  consoles  with  computer- 
driven  visual  and  auditory  displays,  response  processing  and  feedback  capacity, 
etc.  Computer  programs  could  be  designed  to  optimize  learning  and  retention 
of  cognitive  tasks  with  a high  memory  loading,  strategy  and  problem  solving 
tasks,  even  perceptual -motor  tasks.  A centralized  computer  could  serve  a 
number  of  consoles,  yet  preserve  the  individualized  nature  of  instruction. 

This  system  would  Incorporate  well-known  principles  of  computer-aided-instruc- 
tion and  generalized  trainers,  and  it  would  probably  eliminate  the  need  for  the 
large  variety  of  training  media  required  by  the  more  traditional  approach  to 
training.  In  addition,  this  kind  of  system  should  prove  to  be  more  plastic 
and  modifiable,  and  probably  would  reduce  the  time  required  for  training.  This 
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approach  has  the  potential  of  greatly  advancing  modern  training  technology  and 
might  even  prove  to  be  less  expensive  in  the  long  run.  Research  on  the  feasi- 
bility of  such  systems  is  strongly  recommended.  (See  Section  IV  (New  Kinds  of 
Instructional  Media.) 

INTEGRATION  OF  TRAINING  DEVICES  INTO  TRAINING  PROGRAMS 

The  contribution  that  a training  device  makes  to  the  effectiveness  of  a 
training  program  depends  not  only  on  the  characteristics  of  the  training  device 
itself,  but  also  on  the  manner  in  which  it  is  incorporated  into  the  training 
program.  Regardless  of  the  device's  potential  for  training,  if  it  is  utilized 
inappropriately,  its  contribution  to  the  training  program  will  hardly  justify 
its  cost.  Given  a training  device  that  has  been  designed  to  simulate  the  oper- 
ational environment,  what  factors  will  determine  how  it  is  integrated  into  the 
training  program?  While  the  ISD  specifications  provide  some  guidance  with 
respect  to  this  question,  there  appears  to  be  no  procedure  for  determining 
the  answer  to  such  questions  as:  (1)  is  the  training  received  on  a training 
device  more  effective  if  that  training  occurs  at  certain  key  stages  in  the 
training  program,  and  if  so,  how  does  an  ISD  contractor  determine  the  optimal 
stages  for  training  device  use;  (2)  how  does  the  ISD  contractor  establish  what 
skills  need  to  be  trained  prior  to  training  device  use  in  order  that  its  effec- 
tiveness will  be  optimal;  (3)  how  does  the  ISD  contractor  determine  the  kinds 
of  skills  (cognitive,  decision-making,  conmunication,  perceptual -motor,  etc.) 
may  be  most  effectively  trained  by  means  of  a training  device  rather  than  other 
media;  and  (4)  how  does  the  ISD  contractor  establish  the  relationship  that  may 
exist  between  the  kinds  of  skills  the  training  device  is  designed  to  train  and 
the  points  in  the  training  program  at  which  the  training  device  should  be  in- 
serted? Furthermore,  it  would  be  important  for  the  ISD  contractor  to  know  if 
there  were  any  relationships  between  fidelity  of  simulation  and  the  point  in 
the  training  program  when  the  training  device  should  be  introduced.  The  whole 
question  of  how  and  when  the  ISD  contractor  can  optimally  incorporate  a 
training  device  into  a training  program  needs  to  be  investigated. 

INPUTS  TO  TRAINING  DEVICE  DESIGN 

Answers  to  the  above  questions  should  serve  as  important  factors  to  be 
considered  in  the  design  of  new  training  devices.  Ideally,  the  specifications 
for  the  design  of  a training  device  would  include  a complete  enumeration  of  its 
potential  uses,  including  its  potential  for  modification,  the  kinds  of  tasks  it 
is  designed  to  train,  and  the  various  points  within  the  training  program  where 
Its  incorporation  will  be  optimally  effective. 
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SECTION  VI 

INSTRUCTOR  TRAINING  AND  CONTINUITY 

The  questions  raised  in  this  section  are  referenced  to  Data  Item  Descrip- 
tion number  UDI-H-25728  (Training  Device  Instructor/Operator  and  Training 
i-iaterials).  Data  Item  Description  number  UDI-H-25727  (Instructor  Training 
Course  Materials),  and  to  military  specifications  (MIL-T-29053)  Training  Re- 
quirements for  Aviation  Weapon  Systems. 

INSTRUCTOR  QUALIFICATIONS 

The  ISO  specifications  appear  to  be  lacking  in  any  statements  regarding 
Instructor  qualifications.  Surely  the  instructor's  past  experience  with  the 
operational  system  and  his  previous  instructional  experience  are  Important 
factors  that  will  contribute  to  his  success  as  an  instructor  in  the  ISD 
training  program.  Furthermore,  the  Instructor's  personality,  ability  to  re- 
late to  students,  his  speaking  ability,  his  educational  background,  and  his 
motivation  to  be  an  instructor  will  all  contribute  to  his  success  as  an  In- 
structor. None  of  these  factors  are  considered  In  the  ISD  specifications. 

This  means,  essentially,  that  there  are  no  explicit  instructor  selection 
criteria.  This,  of  course,  reflects  the  general  approach  of  the  Navy  to  in- 
struction. Hovever,  It  would  appear  to  be  most  worthwhile  if  a study  could 
be  performed  which  would  demonstrate  unequivocally  the  importance  of  selecting 
instructional  personnel  according  to  criteria  appropriate  for  the  job  they  are 
to  perform. 

INSTRUCTIONAL  PERSONNEL  FOR  TRAINING  DEVICES 

The  ISD  specifications  call  for  the  development  by  the  ISD  contractor  of 
training  programs  for  both  general  instructional  personnel  and  training  device 
Instructional  personnel.  While  the  pertinent  sections  of  the  specs  and  the 
DIDs  appear  fairly  complete  and  wel 1 -formulated , there  are  a few  points  which 
would  appear  to  need  emphasizing.  First,  It  would  appear  essential  that  In- 
structors be  fully  acquainted  with  the  rationale  that  has  led  to  the  specific 
manner  in  which  a training  device  has  been  incorporated  Into  the  ISD  program. 
That  is,  they  should  understand  why  the  trainer  Is  employed  at  certain  points 
in  the  training  program  and  not  at  other  points,  the  general  role  of  the 
trainer  as  It  has  been  Included  In  the  program,  and  the  objectives  which  use 
of  the  trainer  Is  designed  to  accomplish.  In  other  words.  Instructors  should 
be  able  to  see  the  use  of  the  trainer  as  an  Integrated  part  of  the  whole 
program.  Furthermore,  It  would  seem  important  for  all  Instructors,  both  those 
directly  involved  with  training  device  operation  and  those  involved  in  other 
aspects  of  training,  to  be  familiar  with  the  operational  characteristics,  the 
limitations,  the  features  of  the  device  that  involve  a high  degree  of  simula- 
tion, and  the  kind  of  training  exercises  that  will  be  carried  out  in  the 
trainer.  The  point  that  is  being  made  here  is  that  each  instructor,  no  natter 
what  his  specific  responsibility  may  be,  should  have  an  overall  view  of  the 
entire  training  program.  This  will  help  to  prevent  the  program  from  becoming 
segmented  into  parts  that  do  not  dove-tail  smoothly,  and  should  better  enable 
Instructors  to  coninunicate  to  their  students  how  their  respective  parts  of  the 
program  integrate  to  form  the  whole.  Perhaps  some  specifications  along  these 
lines  could  be  developed. 
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INSTRUCTOR  INCENTIVES 

There  Is  little  doubt  that  the  effectiveness  of  an  Instructor  will,  to  a 
large  degree,  depend  upon  the  Instructor's  motivation  In  that  capacity.  The 
Naval  instructor  system,  however,  would  appear  to  offer  relatively  few  Incen- 
tives for  Its  Instructors.  The  Navy's  large  Investment  In  Its  training 
programs,  and  the  importance  of  the  success  of  these  programs  in  maintaining 
an  operational ly-ready  personnel,  would  seem  to  argue  for  an  Instructor  system 
that  Is  both  selective  of  and  rewarding  for  those  personnel  who  participate  in 
It.  In  the  present  system  Instructors  are  only  part-time,  they  do  not  appear 
to  relish  the  duty,  and  they  are  not  continued  In  It  on  a long-term  basis. 
Likewise,  personnel  who  are  designated  to  become  instructors  are  not  selected 
on  the  basis  of  their  potential  expertise  as  instructors.  Surely  this  is  not 
a system  that  Is  designed  to  produce  high-level  Instruction.  Furthermore,  It 
would  appear  to  be  wasteful  of  manpower  resources  due  to  its  lack  of  continuity 
and  the  resulting  need  to  train  new  instructors  for  every  program,  which  pre- 
vents the  accumulation  of  a reservoir  of  experience.  This  system  would  appear 
to  be  unnecessarily  costly,  inefficient,  and  generative  of  a lower  level  of 
instructional  effectiveness  than  could  otherwise  be  obtained  for  less  cost.  It 
Is  strongly  reconmended  that  a study  be  undertaken  which  would  compare  in  de- 
tail the  present  Instructor  system  with  a more  productive  system.  The  study 
would  seek  to  determine  (1)  what  could  be  done  to  modify  the  existing  system  to 
make  It  more  effective,  (2)  what  transitional  steps  would  be  most  direct  and 
least  disruptive  In  making  the  transition  from  the  existing  instruction  system 
to  a new  Instruction  system,  and  (3)  what  incentives  could  be  applied  to  in- 
structor positions  to  make  them  more  desirable  (such  things  as  advancement, 
prestige,  awards  and  special  recognition)? 
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SECTION  VII 


TASK  ANALYSIS,  LISTINGS,  VALIDATION,  AND  SELECTION 

The  questions  raised  In  this  section  are  referenced  primarily  to  Data  Item 
Description  number  UDI-H-25713  (Task  Listings),  Data  Item  Description  number 
UDI-H-25714  (Student  Entry  Level  Report),  Data  Item  Description  number  UDI-H- 
25714  (Task  Selection  Report),  and  to  the  military  specification  number  MIL -T- 
29053,  Training  Requirements  for  Aviation  Weapon  System,  sections  3.1,  3.2, 

3.3,  3.4,  and  6.2. 

THE  TASK  STATEMENT 

According  to  the  specs,  section  3.1,  the  three  pieces  of  information  to  be 
contained  in  a task  statement  are  (1)  "the  action  specified,"  (2)  "the  condi- 
tions of  performance,"  and  (3)  "the  minimum  acceptable  performance  standards." 

No  further  description  or  specification  of  task  statements  are  provided  in  the 
specs.  However,  examples  are  provided  in  section  6.2  of  the  specs.  A close 
study  of  the  examples  provided  Is  somewhat  less  than  satisfying.  The  state- 
ments are  esoteric.  Usually,  no  more  than  one  word  is  used  to  denote  the 
action.  The  conditions  statements  are  brief  to  the  point  of  being  vague.  And, 
the  standards  statements  rarely  contain  anything  approaching  a specific  cri- 
terion for  performance,  and  are  thus  vague  and  probably  useless.  While  this 
evaluation  is  harsh,  it  is  intended  to  be  constructive.  Objective  behavior 
descriptions  are  difficult  and  require  practice  under  expert  guidance.  Since 
the  approach  taken  in  the  specs  is  to  define  task  statements  by  example,  It 
would  seem  very  worthwhile  to  have  an  independent  contractor  develop  a set  of 
Ideal  task  statements  to  be  Included  in  section  6.2.  These  task  statements 
could  be  preceded  by  a brief  description  of  the  behavioral  setting  and  a de- 
tailed examination  of  one  action  sequence  and  its  associated  task  statement. 

The  objective  of  this  would  be  to  call  to  the  reader's  attention  conrnon 
epistemological  and  anthropomorphic  problems  that  arise  in  making  behavioral 
type  statements.  It  would  make  clear  to  the  reader  that  a task  analysis  is 
essentially  a logical  analysis  of  language.  Specifically,  it  Is  suggested  here 
that  section  6.2  be  revised  such  that  It  will  illustrate  for  the  contractor  how 
to  generate  task  statements,  and  will  provide  an  ideal  list  of  task  statements 
for  use  as  examples.  This  should  help  to  improve  the  quality  of  task  listings, 
and  to  bring  about  greater  standardization  from  contractor  to  contractor.  (See 
Section  VIII  (Behavioral  Objectives  and  Hierarchies.) 

ORGANIZATION  OF  TASK  LISTINGS 

According  to  the  specs,  a job  or  position  is  defined  by  Its  responsibility 
areas  and  these  are  further  broken  out  into  missions  and  phases  and  finally 
tasks.  As  illustrated  in  figure  2 of  the  specs,  this  type  of  organization  pro- 
vides the  ''overall  structure  of  a task  listing."  This  type  of  organization 
would  seem  to  raise  several  problems.  First,  as  defined  in  the  specs,  a 
"mission"  Is  a specific  group  activity  with  definitive  beginning  and  end  points, 
and  a specifiable  resultant.  But  this  tends  to  be  confusing  logically  since 
"mission"  is  placed  under  "responsibility  area"  and  therefore  ought  to  be  more 
narrowly  defined.  Responsibility  areas  pertain  to  Individuals,  according  to 
the  specs,  but  missions  may  pertain  to  groups.  Hence,  responsibility  areas 
ought  to  be  subsumed  under  missions.  This  becomes  clear  if  it  is  considered 
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that,  in  the  case  in  a crew  of  one,  a statement  of  missions  would  define  com- 
prehensively responsibility  areas,  and  only  in  this  case  would  missions  fall 
under  responsibility  areas.  Of  course,  in  this  case,  this  would  be  redundant 
organization.  Second,  as  illustrated  in  the  flow  diagram  in  figure  1 in  the 
specs,  and  as  stated  in  sections  3. 1.1. 2 through  3.1 .1.7,  the  format  of  the 
task  listing  structure  is  organized  according  to  the  relationships  superordi- 
nate, ordinate,  and  subordinate.  This  type  of  organization  would  not  appear 
to  be  workable  at  any  level  of  the  task  listing  structure.  The  fact  that  the 
specs  require  that  missions  be  broken  out  into  phases  serves  as  a clear 
acknowledgement  that  the  purely  ordinal  type  organization  would  not  hold  for 
the  mission  levels  of  the  task  listing.  However,  within  each  phase,  the 
enumeration  of  task  statements  is  organized  in  a purely  ordinal  fashion  by  the 
task  listing  process  stipulated  in  the  specs.  Thus,  if  temporal  order  can  be 
taken  Into  account  in  mission  organization,  why  shouldn't  it  be  taken  into 
account  in  the  task  organization?  Because  the  sequential  order  in  which  tasks 
often  have  to  be  performed  may  be  as  important  as  performance  of  the  tasks 
themselves,  it  would  appear  that  the  sequential  relationships  among  tasks 
should  be  preserved  in  the  task  listing.  Furthermore,  it  would  appear  highly 
desirable  to  Identify  in  the  task  listing  any  sequential  dependencies  that 
exist  among  the  tasks  performed  by  different  operators.  For  example,  the  per- 
formance of  a particular  task  by  one  operator  may  depend  on  the  outcomes  of  the 
performance  of  another  task  by  a different  operator,  and  these  two  tasks  may 
have  to  be  performed  in  a tight  time-frame.  This  type  of  coordinated  perform- 
ance of  temporally  dependent  tasks  by  different  operators  may  require  specific 
training  and  should  be  indicated  in  the  task  listing.  Perhaps,  a solution  to 
this  problem  would  involve  merely  a division  of  the  task  listing  process  into 
two  parts:  (1)  an  ordinal  hierarchy  of  task  statements  such  as  is  now  called 
for  by  the  specs,  and  (2)  a kind  of  flow  diagram  showing  for  each  mission  phase 
the  sequential  order  of  tasks  for  each  operator  position.  This  could  be  done 
in  the  following  manner.  A separate  column  would  be  assigned  to  each  operator 
position.  Tasks  would  b e coded  with  numbers  corresponding  to  the  task  state- 
ments they  represent.  The  coded  tasks  would  then  be  listed  in  the  appropriate 
column  in  sequential  order  for  each  position.  Major  time  frames  would  be  indi- 
cated by  horizontal  lines  running  across  columns.  This  would  permit  the  loca- 
tion of  any  tasks  tn  be  performed  by  any  operator  within  the  same  time  frame. 
Sequentially  dependent  tasks  to  be  performed  by  different  operator  positions 
could  be  Indicated  simply  by  arrows  between  columns  connecting  the  sequentially 
dependent  tasks.  Likewise,  arrows  could  be  used  to  indicate  sequentially  de- 
pendent tasks  within  columns  and  dashed  lines  could  be  used  to  indicate  non- 
sequentially  dependent  tasks  within  columns.  It  would  be  necessary  that  all 
tasks  appearing  In  this  flow  diagram  be  of  the  same  ordinal  position.  It  seems 
likely  that  they  should  all  be  subordinate  tasks.  It  is  likely  that  some  sub- 
tasks would  be  repeated  several  times  within  the  flow  diagram  for  particular 
phase.  Construction  of  such  a flow  diagram  would  focus  attention  on  (1)  those 
subtask  sequences  which  must  be  trained  as  sequences  for  any  particular  oper- 
ator position,  (2)  those  subtask  sequences  that  require  coordinated  action  on 
the  part  of  two  or  more  operator  positions  and  must,  therefore,  be  trained  as 
coordinated  action  sequences,  and  (3)  the  relative  frequency  of  occurrence  of 
subtasks  within  and  across  phases  and  thus  help  to  avoid  duplication  in  the 
training  program  and  provide  an  objective  means  of  determining  the  relative 
frequency  of  occurrence  of  subtasks.  In  light  of  the  foregoing  comments,  it 
is  suggested  here  than  an  Independent  study  be  Initiated  to  reconsider  the 
organizational  framework  of  the  task  listing  as  it  is  currently  stipulated  in 
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the  specs.  This  study  would  also  consider  the  merits  of  adding  a time-flow 
type  of  listing  such  as  that  suggested  above  and  develop  a model  which  could 
be  used  to  construct  such  a time-flow  chart. 

TASK  SELECTION 

After  t lie  task  listing  has  been  validated  and  updated,  task  are  often 
classified  according  to  training  requirements.  There  are  five  categories  of 
training  requirements.  These  are:  (1)  "no  training  required,"  (2)  "deferred 
training,"  (3)  "full-scale  training,"  (4)  "review-only  training,"  (5)  "familiar- 
ization-only training."  A task  selection  algorithm  is  offered  in  figure  6 of 
the  specs  as  a suggested  procedure  for  classifying  tasks  according  to  the  five 
categories  of  training  requirements.  The  algorithm  is  based  on  such  factors 
as  the  percentage  of  students  whose  entry  level  behaviors  are  at  criterion  per- 
formance levels,  the  percentage  of  job  holders  that  can  be  expected  to  perform 
the  tasks,  the  frequency  with  which  the  tasks  will  be  performed,  the  critical- 
ity of  the  tasks,  the  beneficence  of  the  tasks,  how  soon  after  arriving  on  the 
job  the  tasks  have  to  be  performed,  and  the  practicality  of  the  on-the-job 
training.  These  criteria  appear  unnecessarily  subjective  and  vague.  For  exam- 
ple, it  Is  probably  rare  that  entry  level  behaviors  can  be  related  in  any  spe- 
cific, or  quantitative  way  to  the  performance  criteria  of  particular  tasks. 

Thus,  a "ballpark"  guess  would  have  to  be  made  regarding  entry  level  behavior. 
Likewise,  judgments  about  the  criticality  and  the  beneficence  of  training  cer- 
tain tasks  must  also  be  subjective  since  neither  of  these  two  words  is  defined 
explicitly.  Judgments  about  frequency  of  occurrence  of  task  performance,  the 
percentage  of  individuals  who  will  be  required  to  perform  each  task,  whether 
performance  of  the  tasks  will  be  required  to  perform  each  task,  whether  per- 
formance of  the  tasks  will  be  required  immediately  after  assuming  the  job 
position,  and  whether  the  task  training  can  be  carried  out  on-the-job,  all 
could  be  determined  objectively,  although  the  current  specs  do  not  require 
this.  Frequency  of  task  performance  could  be  documented.  In  place  of  task 
criticality,  two  other  criteria  could  be  substituted,  namely,  essentialness 
and  risk.  Risk  refers  to  the  potentiality  of  hazard  being  contingent  upon  poor 
task  performance,  and  it  could  be  estimated  in  a reasonably  objective  way  by 
using  a rating  scale.  Essentialness  refers  to  the  necessity  of  Including  a 
task  in  the  training  program  because  it  may  be  prerequisite  to  another  task,  or 
because  of  its  sequential  Interdependence  with  other  tasks,  or  because  it 
occupies  a key  function  in  the  completion  of  a mission.  By  taking  these  three 
dimensions  of  essential  ness  into  account,  fairly  objective  ratings  of  essen- 
tialness could  be  obtained.  This  should  eliminate  any  need  for  the  vague  cri- 
terion of  beneficence.  In  fact,  it  would  probably  be  possible  to  develop  an 
algorithm  for  determining  degree-of-essential ness  using  the  three  dimensions 
mentioned  above.  A similar  approach  could  be  used  In  obtaining  objective 
indices  of  the  other  criteria  that  enter  into  the  task  selection  algorithm.  A 
study  directed  at  generating  an  objectively  based  task  selection  algorithm 
should  be  most  worthwhile.  Considering  that  the  task  selection  algorithm  acts 
as  the  filter  which  feeds  into  the  subsequent  stages  of  the  ISO  process,  a 
well -formulated  task  selection  procedure  could  serve  as  a check  against  Im- 
properly prepared  task  listings.  (See  Tasks  Under  Extraordinary  Conditions  and 
"Hands-on"  Media  and  Tasks,  this  section.) 
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TASK  VALIDATION 

The  specs  require  that  the  initial  task  listing  be  submitted  to  no  less 
than  respondents,  none  of  whom  will  have  been  involved  in  the  initial  task 
listings  in  any  way,  and  that  these  respondents  be  provided  with  the  proper 
forms  and  necessary  instruction  to  carry  out  the  validation  process.  The  cri- 
teria that  the  respondents  will  consider  are  essentially  those  that  enter  into 
the  task  selection  algorithm  described  in  Task  Selection.  Obviously,  if  these 
criteria  were  specified  objectively,  as  argued  in  Task  Selection,  not  only 
would  it  be  easier  for  the  respondent  to  validate  the  task  listing  more  relia- 
bly, but  evaluation  and  Integration  of  the  respondents'  work  would  be  easier. 
However,  even  if  the  criteria  for  use  by  respondents  were  completely  objecti- 
fied, there  remains  the  question  of  how  heavily  their  evaluations  should  be 
weighted  since  they  would  not  have  received  any  ISO  training.  Perhaps  it  would 
be  worthwhile  to  give  a small  number  of  SMEs  a brief  and  highly  specificized 
Introduction  to  ISO  procedures  and  objectives  so  that  they  might  better  be  able 
to  perform  the  task  listing  validation  in  a meaningful  manner.  But  how  many 
SMEs  should  be  included  in  this  group?  The  specs  call  for  no  less  than  ten. 

Is  this  an  optimal  number?  What  kinds  of  problems  are  normally  encountered  in 
carrying  out  task  listing  validation?  Do  the  specs  need  to  be  revised  to  re- 
flect what  has  been  learned  from  ISO  projects  in  which  task  listing  validations 
have  already  been  carried  out?  Interviews  or  questionnaires  with  SMEs  involved 
in  task  listing  and  validation,  as  well  as  contractors,  should  throw  some  light 
on  these  questions.  The  focus  here  would  be  to  determine  what  general  kinds  of 
problems  have  been  encountered  In  different  projects,  suggestions  as  to  how 
those  problems  might  have  been  avoided  by  a different  set  of  specifications, 
and  how  the  various  problems  were  in  fact  resolved.  From  this  sort  of  informa- 
tion, It  should  be  possible  to  deduce  what  revisions  need  to  be  made  to  the 
specs. 

TASKS  UNDER  EXTRAORDINARY  CONDITIONS 

Relating  back  to  the  suggestions  about  criteria  made  in  Task  Selection,  a 
task  to  be  performed  under  extraordinary  conditions  would  be  defined  as  one 
having  (1)  low  frequency  of  occurrence,  (2)  a high  risk  rating,  (3)  a high 
essential  ness  rating,  (4)  hopefully  will  not  be  trained  on  the  job,  (S)  might 
have  to  be  performed  at  any  time  after  assuming  the  job,  (6)  all  personnel  in 
that  job  should  be  prepared  to  perform  it,  and  (7)  it  would  probably  exceed 
entry  level  behavior.  Unless  these  kinds  of  criteria  are  present  in  the  task 
selection  algorithm,  tasks  to  be  performed  under  extraordinary  conditions  may 
not  receive  the  proper  classification  in  the  training  requirements.  While  this 
question  might  well  be  subsumed  under  Task  Selection,  an  independent  study  of 
the  kinds  of  tasks  generated  under  extraordinary  conditions,  together  with  the 
kinds  of  training  these  t a si.  s have  received  in  the  past,  might  reveal  the  kinds 
of  criteria  that  will  be  most  effective  in  selecting  them.  Other  information 
which  could  be  obtained  from  such  a study  might  include  the  training  methodol- 
ogy and  media  most  effective  for  tasks  to  be  performed  under  extraordinary  con- 
ditions. 
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"HANDS-ON"  MEDIA  AND  TASKS 

According  to  the  specs,  preliminary  "hands-on"  media  designation  will  be 
carried  out  after  the  task  listing  has  been  completed,  including  validation  and 
selection.  This  would  appear  somewhat  out  of  place.  Relating  back  to  Task 
Selection,  perhaps  one  of  the  criteria  which  should  be  used  In  the  task  selec- 
tion algorithm  would  be  a "hands-on"  criterion.  This  would  probably  have  the 
effect  of  expanding  the  training  requirements  to  six  or  seven  categories  rather 
than  five.  The  two  additional  categories  would  be  (1)  operational  training, 
and  (2)  simulated  training.  The  implications  of  this  sort  of  change  in  the 
specs  may  be  worth  considering.  For  one  thing,  it  would  mean  that  both  the 
original  group  of  SMEs  and  the  group  of  SMEs  acting  as  respondents  during  task 
validation  would  have  played  a key  role  in  deciding  which  tasks  could  be 
optimally  trained  by  simulation  or  by  actual  operational  experience.  For 
another  thing.  It  might  avoid  elimination  of  those  tasks  which  are,  in  and  of 
themselves,  relatively  trivial  but  which  become  of  some  major  Importance  in  the 
operational  or  simulated  envirorment.  As  a corollary  of  this,  it  may  well  be 
that  some  tasks  might  be  Initially  designated  for  simulator  training  that  might 
otherwise  only  obtain  workbook  training.  In  fact,  the  need  for  certain  scaled- 
4 down  simulator-type  trainers  might  be  identified  in  this  way.  Thus,  this 

question  relates  to  The  New  Training  Device  Concept,  Section  V,  and  to  New  Kinds 
of  Instructional  Media,  Section  IV,  as  well  as  to  Task  Selection  in  this 
section.  Incidentally,  both  simulated  training  and  operational  training  could 
be  subsumed  under  the  training  requirements  calling  for  full  scale  training. 

This  category  would  then  contain  three  subcategories,  namely,  standard  media 
training,  simulated  media  training,  and  operational  training. 
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SECTION  VIII 

BEHAVIORAL  OBJECTIVES  AND  HIERARCHIES 

The  questions  raised  here  are  referenced  primarily  to  Data  Item  Descrip- 
tion number  UDI-H-25717  (Objectives  Hierarchies),  and  to  military  specification 
number  MIL-T-29053,  Training  Requirements  for  Aviation  Weapon  Systems,  sections 
3.7  and  6.3. 

THE  BEHAVIORAL  OBJECTIVE  STATEMENT 

The  criticisms  and  suggestions  made  under  ISD  Process  Segmentation,  Sec- 
tion III,  regarding  the  task  statements  apply  to  some  extent  here.  The  objec- 
tives (task)  hierarchies  given  as  an  example  in  section  6.3  could  probably  be 
improved.  As  was  suggested  in  the  case  of  the  task  listing  given  as  an  example 
in  section  6.2,  an  ideal  task  hierarchy  could  be  developed  and  substituted  in 
section  6.3.  It  would  probably  be  best  if  section  6.2  and  section  6.3  could  be 
developed  together  so  that  the  reader  of  these  specs  could  work  his  way  from 
the  task  listing  through  all  intervening  steps  to  the  task  hierarchy.  In  fact, 
it  would  probably  be  most  beneficial  to  the  contractor  if  all  intervening  steps 
including  decision  algorithms,  interactive  procedures  for  developing  complete 
task  hierarchies,  etc.,  could  be  included  in  the  example.  Extending  this  line 
of  reasoning  to  include  the  entire  ISD  developnental  process,  it  would  seem 
worthwhile  to  develop  an  ideal  example  that  would  start  with  a task  analysis 
on  a scale-down  and  fictitious  mini -operational  system,  and  continue  all  the 
way  through  task  listings,  validation,  and  selection,  to  task  hierarchies 
development,  method  and  media  election,  course  syllabi  development,  training 
support  requirements  analysis,  and  lesson  specification  development.  The  re- 
sulting idealized  example,  then,  would  constitute  a coherent  set  of  materials 
illustrating  the  devel opmental  phase  of  the  ISD  process  in  a direct  and  easily 
understandable  way.  The  operational  system  used  in  the  example  would  need  to 
be  highly  fictitious  to  permit  the  developer  of  this  example  to  create  a system 
just  complex  enough  to  include  only  those  components  essential  to  illustrate 
the  ISD  developmental  procedure.  The  fictitious  mini-system  should  also  have 
some  "sex  appeal,"  that  is,  it  should  have  the  potential  of  holding  the  read- 
er's interest.  It  should  be  familiar,  as  opposed  to  esoteric,  and  perhaps  even 
humorous.  It  should  be  kept  in  mind  that  the  contractor  undertaking  an  ISD 
project  will  prepare  SME  training  materials,  and  that  the  contractor  probably 
will  use  the  example  in  the  specs  for  this  purpose.  So  the  example  provided  in 
the  specs  should  be  prepared  with  an  eye  to  all  potential  users  of  it.  It  has 
not  been  part  of  this  question  to  suggest  that  the  example  be  taken  so  far  as 
to  include  the  preparation  of  reports  specified  in  the  DIDs,  although  this 
might  well  be  done.  In  any  case,  a carefully  constructed  idealized  example  of 
the  ISD  developmental  phase  would  be  a valuable  addition  to  the  specs,  and  it 
is  recommended  here  than  an  independent  study  be  undertaken  to  produce  such  an 
example. 

OBJECTIVES  SELECTION  CRITERIA  AND  ALGORITHM 

Analogous  with  the  task  selection  algorithm,  which  serves  as  a filter  to 
ensure  the  selection  of  significant  tasks  for  training,  the  iterative  process 
for  ensuring  that  objectives  hierarchies  contain  only  instructionally  signifi- 
cant objectives  (sections  3.7.1  .4,  3. 7. 1.6,  3. 7. 1.8,  3.7.1 .9,  3.7.1.10,  and 
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3.7.1.11)  also  serves  as  a kind  of  filter.  However,  in  this  case,  there  is  no 
clear  cut  set  of  criteria.  (1)  In  sections  3. 7. 1.4  and  3. 7. 1.5,  the  specs 
instruct  the  ISO  expert  to  determine  whether  any  decisions  essential  to  the 
performance  of  a task  have  been  omitted  and  to  add  these  decision  tasks  to  the 
components  list.  The  ISO  expert  is  provided  with  a list  of  five  statements 
which  presumably  exhaust  all  possibilities  for  generating  decisions.  Assuming, 
as  is  done  in  decision  theory,  that  an  operator  makes  a decision  each  time  he 
chooses  between  two  or  more  alternatives,  the  five  items  given  in  the  specs  for 
generating  all  possible  decisions  appear  to  be  inappropriate.  (2)  In  section 
3. 7. 1.6,  the  criterion  that  the  ISO  expert  is  instructed  to  use  in  determining 
whether  memorization  is  a significant  element  of  a task  is  simply  whether  an 
average  trainee,  whatever  that  is,  would  be  unable  to  perform  the  task  as  a 
whole  because  he  could  not  remember  which  components  were  to  be  performed,  or 
the  order  of  their  performance.  At  best,  this  criterion  is  vague  and  requires 
an  excessively  subjective  judgment  on  the  part  of  the  ISO  expert.  (3)  In 
section  3. 7. 1.8,  the  ISD  expert  is  instructed  to  determine  whether  there  are 
too  many  subtasks.  The  four  items  provided  for  the  ISD  expert  to  use  in 
determining  this  are  confusing  and  ambiguous.  For  example,  any  subtask  that 
is  found  to  be  a lower  level  component  of  anotiier  subtask  on  the  list  is  sup- 
posed to  be  eliminated.  No  rationale  for  this  is  given,  and  it  is  not  clear 
what  the  ISD  expert  should  do  if  he  judges  the  subtask  to  be  critical  in  its 
particular  position  in  both  places  on  the  list.  The  same  objection  would  be 
raised  with  respect  to  the  next  point  which  stipulates  that  any  subtask  that 
repeats  another  subtask  on  the  list  be  eliminated.  And  the  next  two  items 
clearly  require  subjective  judgments.  One  of  these  instructs  the  ISD  expert 
to  eliminate  a task  if  it  is  not  "necessary"  to  the  accomplishment  of  the  main 
task,  in  his  judgment.  The  other  of  these  items  instructs  the  ISD  expert  to 
eliminate  any  subtask  he  judges  to  be  "trivial."  The  two  words  "necessary"  and 
"trivial"  are  not  defined  explicitly  nor  are  any  criteria  given  by  means  of 
which  "necessary"  or  "trivial"  may  be  determined.  (4)  In  section  3. 7. 1.9,  the 
ISD  expert  is  instructed  to  narrow  the  list  to  the  minimum  set  of  subtasks 
necessary  to  perform  the  terminal  objective.  Again,  the  ISD  expert  is  provided 
with  four  specific  instructions  to  be  used  in  narrowing  the  list.  He  is  first 
instructed  to  eliminate  any  overlapping  subtasks,  although  it  is  not  made  clear 
what  is  objectionable  about  overlapping  subtasks.  In  some  cases,  overlapping 
subtasks  might  even  be  necessary.  Secondly,  the  ISD  expert  is  instructed  to 
eliminate  any  task  that  is  part  of  another  subtask  on  the  list.  Why  this  must 
be  regarded  as  necessarily  objectionable  is  not  at  all  clear.  It  might,  in 
fact,  be  necessary  that  one  subtask  be  part  of  another  subtask  in  order  to  spe- 
cify it  completely.  Thirdly,  the  ISD  expert  is  instructed  to  make  a judgment 
about  which  subtasks  are  not  "essential"  to  performance  of  the  superordinate 
tasks.  The  problem  is,  there  is  no  explicit  definition  of  what  is  meant  by 
"essential,"  and  thus  no  way  to  compare  disagreeing  opinions  of  essentialness 
offered  by  different  SMEs.  Fourthly,  the  ISD  expert  is  instructed  to  group 
"trivial"  subtasks  into  major  logical  categories  and  designate  each  category  as 
a single  subtask.  What  this  means  is  anybody's  guess.  (5)  In  section 
3.7.1.10,  the  ISD  expert  is  given  a formula  for  determining  whether  there  are 
too  few  subtasks.  The  formula  reads,  "If,  after  having  mastered  all  given  sub- 
tasks, the  trainee  would  be  unable  to  perform  the  main  task  without  more  than 
a few  simple  instructions  and  some  coordination  practice,  one  or  more  subtasks 
have  been  omitted."  This  specification  is  accompanied  by  Figure  14  which  shows 
an  algorithm  for  hierarchy  completion.  The  algorithm  states  that  "mastery  of 
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all  subtasks"  plus  "some  minimal  instructions  and  practice"  equals  "mastery  of 
the  major  tasks."  The  problem  with  this  is  that  neither  "minimal  instructions" 
nor  "practice"  are  defined.  As  a general  operating  rule,  it  would  seem  that 
anything  requiring  "instructions"  or  "practice"  should  be  included  as  a sepa- 
rate task  objective  for  training  and  not  left  for  the  whimsical  inspiration  of 
some  instructor  farther  on  down  the  line.  At  any  rate,  if  the  algorithm  for 
hierarchy  completion  is  not  found  to  be  satisfied,  the  ISO  expert  is  instructed 
to  proceed  to  section  3.7.1.11  where  he  is  instructed  to  expand  the  list  of 
subtasks  to  the  minimum  necessary  set,  whatever  that  might  be.  Thus  steps 
3. 7. 1.4  through  3.7.1.11  form  a kind  of  iterative  process  that  is  supposed  to 
result  in  a "complete"  objectives  hierarchy.  This  iterative  process  is  dia- 
gramed in  the  form  of  an  algorithm  in  Figure  12. 

All  in  all,  it  is  clear  what  this  process  is  supposed  to  accomplish,  and 
there  is  little  doubt  that  some  such  process  is  highly  necessary.  The  question 
is:  Will  this  process,  as  it  is  outlined  in  the  specs,  accomplish  the  neces- 
sary and  desired  goal  of  producing  an  objectives  listing  that  is  sufficiently 
complete  to  form  the  basis  of  the  instructional  program?  It  is  hoped  that  the 
above  criticisms  will  indicate  that  there  is  some  doubt  about  the  answer  to 
this  question.  As  in  the  case  for  the  task  selection  process,  this  hierarchy 
selection  process  is  also  a critical  stage  in  the  development  of  an  ISD 
program.  It  would  seem  to  be  sufficiently  important  to  warrant  an  independent 
study  designed  to  develop  a set  of  explicit  objectives  selection  criteria,  and 
to  design  a functionaTly  fine-edged  objective  selection  algorithm. 

ASSESSMENT  OF  DIFFICULTY  OF  BEHAVIORAL  OBJECTIVES 

Whereas,  it  is  the  goal  of  the  task  listing  to  state  precisely  what  an 
operator  must  do  in  order  to  perform  tne  requirements  of  his  jbo,  the  purpose 
of  the  task  components  hierarchy  is  to  provide  an  explicit  statement  of  those 
things  which  a potential  operator  must  be  trained  to  do  in  order  that  he  will 
be  able  to  perform  the  operational  tasks  proficiently.  It  is  implicit  in  this 
distinction  that  operational  tasks  and  training  tasks  (behavioral  objectives) 
are  not  necessarily  identical.  Even  in  the  case  of  high-fidelity  training  de- 
vice simulators,  the  task  selected  for  training  probably  would  not  be  exactly 
the  same  as  the  task  required  in  the  operational  situation. 

In  the  case  of  behavioral  objectives  to  be  achieved  through  training  on 
workbooks,  videotapes,  etc.,  the  question  of  simulation  is  not  even  appropri- 
ate. Much  of  this  sort  of  training  involves  the  learning  of  complex  concepts, 
decision-making  logics,  and  symbolic  stimulus-response  chains  (rules)  that 
serve  as  behavioral  guides,  etc.  These  tasks  constitute  the  "knowledge"  skills 
which  an  operator  must  carry  with  him  to  the  operational  environment. 

Thus,  at  the  task  hierarchy  stage  of  ISD  development,  the  instructional 
design  expert  Is  faced  with  the  question:  Given  the  tasks  specified  in  the 
task  listing,  what  behaviors  must  be  trained  in  order  that  those  tasks  will  be 
performed  proficiently  under  operational  conditions?  Answers  to  this  question 
are  given  In  the  form  of  expanded  task  listings  containing  those  component 
behaviors  judged  to  be  essential  to  wholetask  performance. 
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The  specs  explicitly  recognize  two  types  of  task  components  which  often 
need  to  be  added  at  the  hierarchy  stage,  i.e.,  decisions  (section  3. 7. 1.6), 
However,  even  in  the  case  of  these  two  components,  the  specs  offer  no  objective 
procedure  by  means  of  which  necessary  additional  task  components  may  be  identi- 
fied (this  point  is  treated  in  detail  in  Objectives  Selection  Criteria  and 
Algorithn  of  this  section).  Nor  would  mnemonic  rules  and  decision  components 
appear  to  be  the  only  classes  of  task  components  that  would  need  to  be  added 
at  this  stage  of  ISD  development.  Thus,  the  question  being  raised  here  Is: 

What  pieces  of  information  will  enable  the  ISD  expert  to  generate  the  addition- 
al task  components  needed  to  construct  complete  hierarchies  of  behavioral  ob- 
jectives? 

One  approach  to  answering  this  question  would  be  to  focus  on  the  gap  that 
exists  between  entry  level  behavior  and  criterion  level  behavior,  and  the  kinds 
of  hurdles  the  trainee  must  overcome  in  order  to  cross  that  qap.  Here,  we  are 
concerned  less  with  the  precise  identification  of  the  hurdles  (in  most  cases 
they  probably  could  not  be  identified  completely)  than  with  the  classes  of 
difficulty  that  constitute  the  gap.  There  should  be  a finite  set  of  difficulty 
factors  which  would  comprehensively  characterize  the  problematic  components  In 
ar\y  gap.  By  assessing  the  relative  magnitude  of  each  of  these  difficulty 
factors  for  a particular  gap.  It  should  then  be  possible  to  identify  the  kinds 
of  components  needed  to  bridge  the  gap.  Of  course.  It  would  be  necessary  also 
to  have  some  means  of  specifying  the  magnitude  of  the  gap  as  well.  Hence,  the 
approach  recommended  for  consideration  here  Is  based  on  an  index  of  the  gap 
magnitude  and  the  relative  magnitudes  of  each  of  a finite  set  of  difficulty 
factors. 

Such  an  approach  would  be  based  upon  the  requirement  that,  for  each  task 
selected  for  training,  the  following  items  be  estimated:  (1)  recall  difficulty 
level;  (2)  performance  difficulty  level;  (3)  discrimination  difficulty  level; 
(4)  conceptual  difficulty  level;  and  (5)  the  ratio  of  criterion  level  to  entry 
level  performance.  Let  us  consider  each  of  these  factors  In  some  detail. 

First,  consider  the  ratio  of  criterion  level  to  entry  level  performance. 
The  purpose  of  this  index  would  be  to  specify  the  magnitude  of  the  gap. 

Several  problems  would  be  encountered  with  this  determination.  In  the  first 
place,  entry-level  evaluations  of  prospective  trainees  would  be  only  in  terms 
of  general  qualifications  rather  than  specific  task  component  performance. 

Thus,  given  a task  to  be  trained,  it  is  likely  that  the  ISD  expert  and  SMEs 
should  have  to  estimate  the  entry-level  performances  which  could  be  expected 
on  that  set  of  task  components.  They  probably  would  have  to  base  their  esti- 
mates on  the  general  entry  level  data  available  to  them  plus  their  own  experi- 
ence. The  other  part  of  this  index,  that  Is,  the  criterion  level  performance, 
already  would  be  contained  in  the  task  listing  (though  perhaps  not  for  all  com- 
ponents in  any  given  task  hierarchy).  With  these  two  measurements  In  hand,  the 
ISD  expert  would  then  calculate  the  magnitude  of  the  difference  between  entry 
level  and  criterion  level  performance.  Whether  this  were  expressed  as  a simple 
ratio,  or  a difference,  the  purpose  would  be  to  arrive  at  some  relatively  ob- 
jective estimate  of  the  magnitude  of  the  gap,  i.e.,  how  far  the  trainee  must  be 
moved  by  the  training  program  in  order  to  achieve  criterion  level  performance. 
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The  magnitude  of  the  gap,  however,  would  not  be  very  meaningful  in-and-of 
itself.  Rather,  it  would  have  to  be  understood  in  terms  of  the  overall  diffi- 
culty of  getting  across  the  gap.  For  example,  a very  large  gap  may  be  found  to 
exist  between  entry  level  and  criterion  level  for  a particular  task  (or  task 
component),  but,  if  the  difficulty  of  achieving  criterion  performance  were  very 
low,  relatively  little  training  might  be  required.  Conversely,  a small  gap 
associated  with  great  difficulty  might  require  extensive  training.  Hence,  the 
magnitude  of  the  gap  would  indicate  only  how  far  the  trainee  must  be  brought 
to  achieve  criterion  level  performance.  It  would  not  indicate  the  rate  at 
which  the  trainee  might  achieve  that  performance  level.  The  latter  would  be 
indicated  by  the  four  difficulty  factors. 

The  magnitude  of  the  four  difficulty  factors  could  be  estimated  by  SMEs 
who  might  use  a standard  rating  scale  associated  with  an  explicit  definition 
of  each  factor.  For  example,  in  estimating  the  magnitude  of  the  recall  diffi- 
culty level  for  a particular  task,  SMFs  might  first  ask  themselves  tlie  follow- 
ing question:  What  information  bits  and  bit  sequences  must  ttie  operator  be 
able  to  recall  (either  verbally  or  functionally)  in  order  to  perform  this  task 
which  has  been  selected  for  training?  After  having  broken  down  the  task  into 
Its  recall  components,  the  SMEs  would  assign  to  each  component  a number  from 
the  rating  scale  for  recall  difficulty.  If,  for  example,  the  rating  scale  had 
five  categories,  the  number  five  would  be  assigned  to  a component  judged  to  be 
very  difficult  to  recall,  and  so  on  for  the  other  categories  on  the  scale. 
Whatever  else  "great  difficulty  to  recall"  may  mean  to  SMEs,  they  could  be  in- 
structed by  the  ISD  expert  to  base  their  estimate  of  recall  difficulty  on  the 
number  of  rehearsal  trials  they  think  would  be  regu i red  for  perfect  recal 1 . 

This  Instruction  would  define  the  upper  end  of  the  rating  scale.  The  remaining 
categories  on  the  scale  could  be  defined  in  a similar  manner.  By  systematical- 
ly applying  this  scale  to  each  task  component,  the  degree  of  recall  training 
required  for  each  component  would  be  specified  (some  procedure  for  averaging 
individual  SME  ratings  would  need  to  be  worked  out,  of  course).  This  procedure 
would  not  only  identify  task  components  that  require  recall  training,  but  would 
Indicate  the  degree  of  training  required. 

The  next  factor,  performance  difficulty  level,  would  be  estimated  in  much 

the  same  way  using  the  same  sort  of  rating  scale.  First,  the  SMEs  would  break 

the  task  dov*i  into  its  performance  components,  i.e.,  each  group  of  overt 

actions  into  which  the  task  may  be  subdivided.  Note  that  not  all  task  compon- 

ents requiring  training  would  involve  overt  actions  in  the  operational  situa- 
tion. After  identification  of  task  components  involving  overt  actions,  SMEs 
would  estimate  the  performance  difficulty  of  each.  They  could  base  their  esti- 
mates of  difficulty  on  the  answer  to  the  following  question:  How  difficult  is 
It  to  perform  this  task  component  in  the  operational  environment  even  after  It 
has  been  learned  perfectly?  This  would  define  the  upper  end  of  the  rating 
scale. 

The  Idea  here  is  to  separate  learning  difficulty  from  performance  diffi- 
culty. It  should  be  pointed  out  that  the  recall  task  components  identified 
above  would  also  be  considered  under  performance  task  components.  The  enphasls 
in  estimating  recall  difficulty  is  on  learning,  but  the  same  task  components 
need  to  be  evaluated  from  the  point  of  view  of  performance  difficulty  as  well. 
For  example,  if  a task  component  requiring  recall  is  found  to  be  exceedingly 
difficult  to  perform  in  a simulated  environment  even  after  It  has  been  learned 
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perfectly  elsewhere,  the  best  course  of  action  probably  would  be  to  provide  a 
memory  aid,  or  to  automate  this  component  by  means  of  an  on-board  computer  with 
associated  display,  thus  substituting  a computation  component  in  place  of  the 
recall  component.  The  same  considerations  would  be  especially  evident  ftit* 
purely  motor  components.  If  the  perfomiance  difficulties  of  motor  components 
were  excessively  high,  perhaps  the  device  to  be  operated  could  be  modified. 

Even  If  this  were  not  a possibility,  an  Important  source  of  difficulty  would 
have  been  Identified  and  properly  attributed  to  purely  performance  limitations 
rather  than  confusing  them  with  learning  difficulties. 

The  next  difficulty  factor,  discrimination  difficulty,  refers  to  the 
relative  ability  of  operators  to  distinguish  among  the  Informationally  signifi- 
cant elements  of  perceptual  displays.  These  perceptual  elements  may  not  be 
limited  to  any  single  sensory  modality.  They  may  include  tactile  displays 
which  must  be  discriminated  with  the  fingertips,  motion  or  balance  cues  which 
must  be  discriminated  by  means  of  the  senses  of  equilibrium,  acoustic  signals 
which  must  be  differentiated  by  the  auditory  system,  and  visual  cues  necessary 
to  read  the  various  cockpit  displays  and  to  perform  the  visually-guided  air- 
craft maneuvers.  As  before,  the  SMEs  could  use  a rating  scale  as  the  basis  for 
estimating  discrimination  difficulty.  For  each  discrimination  task  component 
Identified,  SMEs  would  be  Instructed  to  ask  themselves  the  following  question: 
How  much  practice  Is  necessary  in  order  to  achieve  perfect  discrimination  for 
this  component?  The  answer  to  this  question  could  be  translated  Into  the  form 
of  a number  from  the  rating  scale.  For  example.  If  an  Infinite  amount  of 
practice  would  result  in  only  poor  discrimination,  the  number  at  the  top  of 
the  scale  would  be  assigned  to 'this  task  component.  On  the  other  hand.  If  per- 
fect discrimination  could  be  achieved  with  a moderate  amount  of  practice,  the 
number  at  the  middle  of  the  scale  would  be  assigned  to  this  component.  This 
number  would  represent  a learning  difficulty.  However,  a very  high  discrimina- 
tion difficulty  rating  would  Indicate  more  than  simply  a training  problem.  For 
example.  If  a display  were  poorly  Illuminated,  an  accurate  reading  of  the 
callibrated  scale  on  the  display  might  exceed  the  limits  of  human  visual 
acuity.  In  such  a case,  no  amount  of  training  would  overcome  this  difficulty. 
Thus,  a high  discrimination  difficulty  rating  probably  should  be  taken  to  Indi- 
cate a needed  modification  In  some  aspect  of  the  Informational  display.  In 
fact,  if  criterion  performance  on  this  task  component  requires  a high  degree  of 
accurate  discrimination,  especially  If  It  must  be  achieved  quickly,  even  a 
moderate  level  of  discrimination  difficulty  might  Indicate  the  need  for  a modi- 
fication of  the  display. 


It  should  be  evident  that  both  discrimination  difficulty  and  performance 
difficulty  are  probably  the  result  of  poor  human  factors  engineering,  and  both 
may  be  Indicative  of  needed  modifications  of  the  operational  device^  A high 
discrimination  difficulty  would  Indicate  the  need  for  a modification  In  the 
display,  while  a high  performance  difficulty  on  a manual  task  component  would 
Indicate  a need  for  modification  In  the  operator's  controls. 

However,  the  existence  of  either  of  these  two  difficulty  factors  should 
not  be  taken  as  an  automatic  indication  of  a need  for  device  modification. 

That  judgment  always  should  be  made  within  the  context  of  the  ratio  of  cri- 
terion level  to  entry  level  performance,  plus  the  risk  factor  estimated  during 
task  selection.  The  decision  formula  would  be:  If  the  ratio  of  criterion 
level  to  entry  level  Is  large,  and  If  risk  Is  high,  substantial  discrimination 
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or  performance  difficulties  should  require  device  modification.  On  the  other 
hand,  if  the  ratio  of  criterion  level  to  entry  level  is  moderate,  and  if  risk 
is  low,  even  a significant  performance  or  discrimination  difficulty  probably 
Would  not  require  a device  modification.  In  this  case,  it  probably  fcGUld  be 
more  appropriate  to  minimize  the  discrimination  or  performance  difficulty  by 
employing  some  form  of  specialized  training,  even  if  it  means  a reduction  In 
the  acceptable  level  of  performance. 

The  fourth  difficulty  factor  to  be  considered  is  conceptual  difficulty. 
This  is  a purely  training  type  difficulty.  The  conceptual  difficulty  of  a task 
component  may  be  defined  in  terms  of  the  extensiveness  of  training  required 
for  trainees  to  learn  the  concept  involved  and  to  be  able  to  correctly  apply 
it.  As  in  the  case  of  the  other  difficulty  factors,  SMEs  would  first  analyze 
a task  selected  for  training  to  determine  whether  it  contained  any  components 
that  were  identifiable  as  concepts.  It  would  then  be  necessary  to  determine 
the  logical  nature  of  each  concept.  Concept  types  may  be  differentiated  by 
means  of  certain  specific  logical  operations.  As  these  operations  become  more 
complex,  the  concepts  become  more  difficult  for  humans  to  learn.  Five  concept 
types  (each  type  has  its  complementary  type)  may  be  distinguished  logically. 
Progressing  from  least  complex  to  most  complex,  the  five  concept  types  are: 
affirmation,  conjunction,  inclusive  disjunction,  conditional,  and  bi-condition- 
al. It  has  been  found  that  concepts  based  on  affirmation  alone  are  more  easily 
learned  than  concepts  based  on  coigunction,  inclusive  disjunction,  and  condi- 
tional relations.  Concepts  based  on  bi -conditional  relations  are  the  most 
difficult  for  humans  to  learn.  Thus,  three  levels  of  conceptual  difficulty 
have  been  Identified,  with  affirmational  concepts  the  easiest  and  bi -condition- 
al concepts  the  most  difficult.  Concepts  involving  conjunction,  inclusive  dis- 
junction, and  conditional  relations  appear  to  be  of  intermediate  difficulty. 
Determination  of  the  relative  difficulties  of  the  concepts  in  the  intermediate 
level  group  must  be  left  to  future  research.  The  point  here  is  that  any 
training  program  designed  for  humans  involves  the  learning  of  concepts.  And, 
since  concept  difficulty  depends  on  the  logical  relations  that  compose  con- 
cepts, a straightforward  procedure  is  suggested  for  determining  the  conceptual 
difficulty  factor  for  task  components.  After  SMEs  identify  the  conceptual 
components  in  a task,  they  would  then  simply  determine  which  set  of  logical 
operations  define  each  task  component.  Once  the  logical  operations  have  been 
| determined,  the  type  of  concepts  have  also  been  established.  And  these  types 

• fall  into  three  classes  of  relative  difficulty.  Hence,  the  only  problem  facing 

SMEs  would  be  determination  of  the  logical  operations. 

The  five  sets  of  logical  o|*rations  corresponding  to  the  five  concept 
types  are:  (1)  affirmation  - "all  things  that  are  'A'  belong  to  the  same 
class";  (2)  conjunction  - "all  things  that  are  both  'A'  and  * B ’ are  members  of 
the  same  class11;  (3)  inclusive  disjunction  - "all  things  that  are  either  'A' 
or  1 B * are  members  of  the  same  class";  (4)  condi tional  - "the  relationship  be- 
tween events  'A'  and  ' B*  is  such  that  instances  of  'B1  occur  only  if  instances 
of  event  'A'  have  occurred,  but  not  the  converse";  (5)  bi-conditional  - "if  an 
event  is  an  Instance  of  'A',  then  it  must  be  accompanied  by  an  event  that  is 
an  instance  of  ’B’,  and  vice  versa." 
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Some  concrete  examples  of  these  conceptual  relations  are:  (1)  affirma- 
tion - "all  red  lights  signal  danger";  (2)  conjunction  - "if  both  red  Tight  1 
and  red  light  2 illuminate  simultaneously,  system  malfunction  has  occurred"; 

(3)  Inclusive  disjunction  - "if  either  red  light  1 or  red  light  2 illuminate, 
system  malfunction  has  occurred";  (4)  conditional  - "the  illumination  of  red 
light  2 indicates  system  malfunction  only  if  red  light  1 was  illuminated, 
otherwise,  illumination  of  red  light  2 means  something  different";  (5)  bi- 
conditional - "illumination  of  red  light  1 indicates  system  mal function”Tf , 
and  only  if,  illumination  of  red  light  2 has  occurred;  and  illumination  of 
red  light  2 may  indicate  system  malfunction  if,  and  only  if,  illumination  of 
red  light  1 has  occurred."  It  should  be  clear  from  the  foregoing  examples 
that  most  rules  are,  in  fact,  concepts. 

It  is  recommended  here  that  SMEs  estimate  the  difficulty  of  such  concepts 
by  first  identifying  the  type  of  concept  involved,  and  then  by  determining  the 
decisions  or  actions  which  must  be  taken  on  the  basis  of  that  concept.  With 
these  two  elements  in  mind,  SMEs  should  be  able  to  use  a rating  scale  of  diffi- 
culty to  establish  difficulty  levels  of  conceptual  task  components. 

The  result  of  the  foregoing  analyses  would  be  a list  of  task  components 
each  of  which  would  have  associated  with  it  five  numerical  indices:  (1)  recall 
difficulty  level,  (2)  performance  difficulty  level,  (3)  discrimination  diffi- 
culty level,  (4)  conceptual  difficulty  level,  and  (5)  ratio  of  criterion  level 
to  entry  level  performance.  These  factors  form  a cluster  which  specifies,  for 
each  task  component,  how  far  the  trainee  must  be  moved  by  the  training  program, 
and  what  types  of  difficulty  the  task  hierarchy  must  be  designed  to  minimize. 
This  information  should  make  it  possible  for  the  ISD  expert  to  determine  the 
magnitude  of  importance  of  each  training  objective  at  the  time  he  constructs 
the  task  hierarchy.  Furthermore,  this  information  should  make  it  possible  for 
the  ISD  expert  to  construct  an  hierarchy  for  a set  of  task  components  which 
will  (1)  avoid  unnecessary  training  on  any  task  component,  (2)  specify  the 
particular  type  of  training  needed  in  order  to  overcome  the  major  sources  of 
difficulty  associated  with  any  task  component,  and  (3)  organize  the  task  com- 
ponents into  an  hierarchy  which  will  take  into  account  the  skill,  conceptual, 
and  temporal  interdependencies  of  task  components.  The  latter  point  will  be 
discussed  further  In  this  section. 

This  system  of  task  component  analysis  appears,  on  the  surface,  to  be 
somewhat  more  complex  than  that  recommended  in  the  specs.  This  is  probably  not 
entirely  correct.  The  apparent  complexity  of  the  approach  recommended  here,  as 
compared  with  that  recommended  in  the  specs,  is  probably  due  to  the  fact  that 
here  an  attempt  has  been  made  to  objectify  the  kinds  of  considerations  that 
probably  would  be  present  in  any  task  components  analysis.  However,  the 
specifics  of  the  approach  recommended  here  need  to  be  subjected  to  further 
theoretical  and  empirical  examination. 

Some  examples  of  questions  that  need  further  examination:  (1)  are  the 
difficulty  factors  specified  in  the  above  approach  both  the  necessary  and 
sufficient  ones  for  valid  characterization  of  the  problematic  elements  of  task 
components;  (2)  what  special  problems  might  SMEs  encounter  in  usino  such  a 
system  as  that  recommended  above;  (3)  what  system  will  afford  the  most  workable 
and  effective  application  of  findings  from  research  on  conceptual  operations, 
concept  formation,  and  concept  learning;  and  (4)  how  may  such  an  approach  as 
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the  one  recommended  here  be  incorporated  least  disruptively  into  the  specs? 
These  are  but  a few  of  the  questions  "aised  by  the  approach  suggested  here. 
Because  of  the  potential  benefit  of  such  an  approach  to  the  ISD  process,  one 
or  more  Independent  studies  In  this  area  would  appear  warranted. 

TASK  HIERARCHY  ORGANIZATION 

According  to  the  specs,  it  is  the  task  hierarchy  which  specifies  the 
organization  of  what  is  to  be  trained.  However,  the  kind  of  organization 
called  for  in  the  specs  would  not  appear  to  be  one  that  is  optimal  for  syllabi 
development.  The  hierarchy  organization  laid  out  in  the  specs  follows  a simple 
reduction  logic,  i.e.,  objectives  are  arranged  according  to  the  relations 
superordinate,  ordinate,  and  subordinate.  The  only  pieces  of  information  that 
this  organization  contains  are  (1)  the  list  of  objectives  to  be  trained,  and 
(2)  the  ordinal  relation  among  those  objectives.  However,  as  pointed  out  in 
Assessment  of  Difficulty  of  Behavioral  Objectives  above,  the  reduction  type 
hierarchy  does  not  contain  the  sort  of  information  which  would  make  possible 
(1)  the  avoidance  of  unnecessary  training  on  task  components,  (2)  any  indica- 
tion of  the  type  of  training  needed  in  order  to  overcome  the  major  sources  of 
difficulty  associated  with  task  components,  and  (3)  the  kinds  of  relationships 

among  task  components  which  must  be  learned  (as  opposed  to  those  which  are 

purely  logical)  such  as  the  skill,  conceptual,  and  temporal  interdependencies 
among  task  components.  The  information  needed  for  items  (1)  and  (2)  was 
thoroughly  discussed  in  question  above.  It  Is  the  purpose  of  this  question  to 

consider  the  information  in  item  (3),  i.e.,  to  consider  an  alternative  organi- 

zational structure  for  task  hierarchies. 

The  general  strategy  of  the  approach  recottmended  here  is  to  first  identify 
the  task  components  which  require  training,  then,  to  establish  the  type  and 
extensiveness  of  training  required  to  achieve  criterion  level  performance  for 
each  component,  and  finally,  to  arrange  these  components  into  an  hierarchical 
organization  such  that  actual  dependencies  among  components  are  preserved,  and 
difficulty  factors  are  overcome  at  the  component  level  of  organization.  Given 
that  a difficulty  analysis  has  been  performed  and  the  result  is  a list  of  task 
components,  together  with  a cluster  of  difficulty  factors  for  each  component, 
then  the  components  may  be  organized  into  an  hierarchy  by  means  of  the  follow- 
ing rules:  (1)  difficulty  rule  - arrange  the  hierarchical  organization  such 
that,  the  task  components  wlrleli  appear  at  any  organizational  level  are  ordered 
according  to  the  extensiveness  of  training  required  to  bring  each  component  up 
to  its  criterion  level  so  that,  when  the  individual  components  are  combined  at 
the  next  higher  level  of  the  hierarchy,  performance  on  each  individual  compon- 
ent in  the  combination  is  at  criterion  level;  (2)  sequential  rule  - the  sequen- 
tial relationships  among  temporally  dependent  components  1 s to  be  represented 
as  a compound  component  at  a higher  level  of  hierarchical  organization  than  the 
Individual  components,  and  the  compound  enumerates  the  sequential  order  to  be 
learned  (If,  In  addition  to  performing  the  compound  action  sequence,  practice 
on  some  mnemonic  device,  such  as  verbal  recall  of  the  sequence,  is  judged 
necessary,  this  may  be  Included  as  a separate  compound  component  at  the  same 
hierarchical  level  of  organization);  (3)  skill  rule  - if  the  relationship  be- 
tween any  two  components  is  such  that  competent  performance  on  one  is  a pre- 
requisite for  performance  on  the  other,  the  prerequisite  component  should 
appear  before  the  dependent  component  In  the  organizational  hierarchy  vrtiether 
or  not  the  prerequisite  component  is  more  or  less  difficult  than  the  dependent 
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component;  (4)  concept  rule  - organize  the  components  of  a concept  according 
to  their  logical  operations,  such  that  the  least  complex  conceptual  components 
(class  concepts)  appear  first  in  the  hierarchy  and  the  more  complex  conceptual 
components  (relational  concepts)  appear  at  higher  levels  of  the  hierarchy. 

These  rules  for  hierarchy  construction  are  somewhat  overlapping.  For 
example,  the  difficulty  rule  and  the  concept  rule  are  redundant  if,  and  only 
If,  the  compound  of  task  components  is  conceptual.  Otherwise,  the  concept  rule 
would  not  apply,  but  the  difficulty  rule  would.  Likewise,  if  the  compound  of 
task  components  Is  sequential,  the  sequential  rule  and  the  skill  rule  would  be 
redundant.  Otherwise,  the  skill  rule  would  apply. 

Syllabi  developed  to  incorporate  the  hierarchical  relations  among  task 
components  generated  by  these  four  rules  should  maximize  learning  by  (1)  re- 
ducing to  a minimum  component  difficulty  at  each  hierarchical  level,  (2)  en- 
suring criterion  performance  of  each  component  in  a compound  at  each  hier- 
archical level,  (3)  ensuring  the  component  completeness  of  compounds  at  each 
hierarchical  level,  (4)  preserving  conceptual,  temporal,  and  skill  dependencies 
among  components  throughout  hierarchical  chains,  and  (5)  providing  a behavior- 
ally  meaningful  procedure  for  integrating  compounds  of  entirely  different  kinds 
of  components.  Furthermore,  since  difficulty  analysis  of  components  already 
would  have  been  carried  out  prior  to  hierarchy  construction,  the  kind  of 
training,  as  well  as  the  extensiveness  of  it,  would  have  been  made  evident  for 
each  component. 

It  would  appear  that  the  merits  of  some  such  organizational  structure  for 
task  hierarchies  would  warrant  a full-scale  study  of  its  potential  for  inclu- 
sion in  the  ISD  process.  Perhaps,  as  a way  of  determining  its  advantages  and 
disadvantages,  a specific  system  of  very  limited  scope  could  be  used  to  gener- 
ate a task  analysis  on  the  basis  of  which  several  approaches  to  hierarchy  con- 
struction (that  suggested  here  as  compared  with  the  reduction  approach  con- 
tained in  the  specs)  could  be  evaluated  empirically.  Since  there  would  seem 
to  be  little  question  about  which  of  the  two  types  of  approaches  would  lead  to 
more  efficient  training,  the  main  question  to  be  answered  by  such  a study 
would  be:  Can  the  more  appropriate  form  of  hierarchy  construction  be  carried 
out  successfully,  if  not  efficiently,  by  SMEs? 

CONCEPTUAL  ANALYSIS  AND  DIFFICULTY 

The  pertinence  of  operational  analysis  of  concepts,  and  concept  learning 
difficulty,  to  the  ISD  process  has  been  described  in  detail  in  this  section. 
Thus,  the  question  here  has  been  included  to  focus  attention  on  two  problems 
which  emerged  In  the  previous  questions.  Because  of  the  special  nature  of 
these  problems,  and  because  they  both  Involve  conceptual  analysis  and  learning. 
It  is  recommended  that  an  independent  study  be  undertaken  to  explore  them 
further.  The  two  problems  are:  (1)  the  relationship  between  concept  learning 
difficulty  and  conceptual  logical  operations,  and  (2)  the  usefulness  of  con- 
ceptual logical  operations  as  a basis  for  task  component  hierarchy  organiza- 
tion. The  findings  of  a study  directed  at  these  two  problems  could  provide 
Important  Inputs  to  the  studies  suggested  under  the  two  previous  questions. 
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